Skip to content

test(prose): claims assert consequences, not what was displayed#556

Open
leeovery wants to merge 1 commit into
prose-tests/discussion-conversation-stubfrom
prose-tests/consequence-claims
Open

test(prose): claims assert consequences, not what was displayed#556
leeovery wants to merge 1 commit into
prose-tests/discussion-conversation-stubfrom
prose-tests/consequence-claims

Conversation

@leeovery

@leeovery leeovery commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • P4c was written and not applied. The runs then showed why it matters. root-cause-validation-clean-verdict failed only on "offers the choice and waits" and "announces that validation is running". implementation-picks-first-task came back FLAKY on "asks the question, gathering the answer" — evidenced in one run's narrative, absent from the other, with an identical world both times. Display claims don't merely fail, they fail intermittently, which is worse than failing.
  • Every display-only claim in the corpus is rewritten as something the record can prove:
was now
"offers the choice and waits" the scripted yes was consumed, so the run arm was taken — a skip would have returned with no dispatch
"announces running, then dispatches exactly one agent" one dispatch recorded, one report written, no second of either
"asks the question, gathering without acting on it" the answer was consumed there, and the entry wrote and committed no setup document
"renders the phase note and emits it as produced" renders it through the engine with the verb Starting — the render is an engine call; the emission isn't
  • Swept the whole corpus rather than only the two failing cases, so the rule is applied uniformly.

No prose touched — test authoring only.

Test plan

  • npm test green: 1699 tests, 0 fail. Corpus valid.
  • Both previously-failing cases should now be judged on evidence that exists — worth re-running to confirm, though a PASS then means the claims are provable rather than that anything in the prose changed.

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556 👈 current
  13. feat(prose-tests): record everything the agents do, results included #557
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571

@leeovery
leeovery force-pushed the prose-tests/discussion-conversation-stub branch from f83df8e to 9d3d1b3 Compare July 26, 2026 13:43
P4c was written and not applied. The runs then demonstrated why it
matters: root-cause-validation failed only on "offers the choice and
waits" and "announces that validation is running", and
implementation-picks-first-task came back FLAKY on "asks the question,
gathering the answer" — evidenced in one run's narrative, absent from
the other, with an identical world both times. Display claims don't
merely fail; they fail intermittently, which is worse than failing.

Each is rewritten as something the record can prove:

- "offers the choice and waits" → the scripted yes was consumed, so the
  run arm was taken; a skip would have returned with no dispatch
- "announces running, dispatches exactly one agent" → one dispatch
  recorded, one report written, no second of either
- "asks the question, gathering without acting" → the answer was
  consumed there, and no setup document was written or committed by the
  entry
- "renders the phase note and emits it as produced" → renders it through
  the engine with the verb Starting; the render is an engine call, the
  emission is not

No prose touched — this is test authoring only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leeovery
leeovery force-pushed the prose-tests/consequence-claims branch from b55f4f4 to 4790742 Compare July 26, 2026 13:44
This was referenced Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant